We are using a two-component hurdle model: first, the model predicts whether a disease will be present (binary), and if present, it predicts the case count (integer). Here we compare the results of a boosted tree model to our baseline model.

Disease Status

disease status confusion matrix
.metric desc model full_model
accuracy proportion of the data that are predicted correctly baseline 0.85
xgboost 0.96
kap similar measure to accuracy(), but is normalized by the accuracy that would be expected by chance alone and is very useful when one or more classes have large frequency distributions. baseline 0.44
xgboost 0.87
sens the proportion of positive results out of the number of samples which were actually positive. baseline 0.99
xgboost 0.98
spec the proportion of negative results out of the number of samples which were actually negative baseline 0.35
xgboost 0.88
disease status confusion matrix by taxa
.metric model birds buffaloes camelidae cats cattle cervidae dogs equidae hares/rabbits sheep/goats swine
accuracy baseline 0.85 0.76 0.780 0.78 0.86 0.730 0.79 0.91 0.86 0.86 0.87
xgboost 0.94 0.95 0.960 0.96 0.95 0.950 0.94 0.97 0.96 0.96 0.96
kap baseline 0.41 0.19 0.120 0.39 0.56 0.066 0.52 0.43 0.20 0.45 0.43
xgboost 0.82 0.89 0.880 0.91 0.88 0.890 0.88 0.86 0.86 0.87 0.88
sens baseline 0.98 1.00 1.000 1.00 0.99 1.000 0.99 0.99 0.99 0.99 0.99
xgboost 0.97 0.97 0.980 0.97 0.98 0.960 0.96 0.99 0.99 0.98 0.98
spec baseline 0.33 0.14 0.086 0.33 0.49 0.049 0.48 0.31 0.14 0.36 0.33
xgboost 0.83 0.91 0.900 0.94 0.89 0.940 0.92 0.84 0.84 0.88 0.89
disease status confusion matrix by continent
.metric model Africa Americas Asia Europe NA Oceania
accuracy baseline 0.84 0.82 0.85 0.87 0.95 0.930
xgboost 0.95 0.96 0.96 0.95 NA 0.990
kap baseline 0.47 0.38 0.46 0.46 0.56 0.100
xgboost 0.87 0.90 0.88 0.83 NA 0.920
sens baseline 0.99 0.99 0.99 0.99 1.00 1.000
xgboost 0.97 0.98 0.98 0.98 NA 1.000
spec baseline 0.40 0.30 0.37 0.37 0.42 0.058
xgboost 0.89 0.91 0.88 0.84 NA 0.890
disease status direction change confusion matrix
.metric desc model full_model
accuracy proportion of the data that are predicted correctly baseline 0.850
xgboost 0.960
kap similar measure to accuracy(), but is normalized by the accuracy that would be expected by chance alone and is very useful when one or more classes have large frequency distributions. baseline 0.051
xgboost 0.480
sens the proportion of positive results out of the number of samples which were actually positive. baseline 0.470
xgboost 0.540
spec the proportion of negative results out of the number of samples which were actually negative baseline 0.680
xgboost 0.790
Note there are baseline cases where disease status is positive but cases are NA, which are imputed in the model as 0.
disease status direction change confusion matrix by taxa
.metric model birds buffaloes camelidae cats cattle cervidae dogs equidae hares/rabbits sheep/goats swine
accuracy baseline 0.850 0.760 0.780 0.78 0.860 0.730 0.790 0.91 0.860 0.860 0.870
xgboost 0.940 0.950 0.960 0.96 0.950 0.950 0.940 0.97 0.960 0.960 0.960
kap baseline 0.066 0.025 0.032 0.05 0.026 0.015 0.051 0.10 0.036 0.057 0.056
xgboost 0.330 0.570 0.600 0.66 0.460 0.640 0.580 0.45 0.490 0.490 0.480
sens baseline 0.450 0.580 0.580 0.58 0.420 0.560 0.500 0.50 0.470 0.490 0.480
xgboost 0.490 0.560 0.570 0.59 0.530 0.590 0.580 0.55 0.560 0.530 0.530
spec baseline 0.690 0.660 0.660 0.67 0.670 0.650 0.670 0.71 0.670 0.680 0.680
xgboost 0.740 0.830 0.830 0.85 0.780 0.860 0.830 0.78 0.790 0.790 0.790
disease status direction change confusion matrix by continent
.metric model Africa Americas Asia Europe NA Oceania
accuracy baseline 0.840 0.820 0.850 0.87 0.95 0.930
xgboost 0.950 0.960 0.960 0.95 NA 0.990
kap baseline 0.038 0.023 0.055 0.09 0.26 0.027
xgboost 0.490 0.540 0.480 0.42 NA 0.570
sens baseline 0.460 0.470 0.470 0.47 0.58 0.600
xgboost 0.520 0.540 0.550 0.53 NA 0.610
spec baseline 0.670 0.660 0.680 0.69 0.74 0.680
xgboost 0.790 0.810 0.790 0.77 NA 0.810

Cases

Here we evaluate the subset of the training data with positive case counts

cases model stats

## # A tibble: 6 x 4
##   model    .metric .estimator  .estimate
##   <chr>    <chr>   <chr>           <dbl>
## 1 baseline rmse    standard   171773.   
## 2 xgboost  rmse    standard   224808.   
## 3 baseline rsq     standard        0.669
## 4 xgboost  rsq     standard        0.494
## 5 baseline mae     standard     1580.   
## 6 xgboost  mae     standard     2039.
cases residuals
cases residuals by taxa
cases residuals by continent